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Abstract.  Atomicity  (or  linearizability)  is  a  commonly  used  consistency 
criterion  for  distributed  services  and  objects.  Although  atomic  object  im¬ 
plementations  are  abundant,  proving  that  algorithms  achieve  atomicity 
has  turned  out  to  be  a  challenging  problem.  In  this  paper,  we  initiate 
the  study  of  systematic  ways  of  verifying  distributed  implementations 
of  atomic  objects,  beginning  with  read/write  objects  (registers).  Our 
general  approach  is  to  replace  the  existing  operational  reasoning  about 
events  and  partial  orders  with  assertional  reasoning  about  invariants  and 
simulation  relations.  To  this  end,  we  define  an  abstract  state  machine 
that  captures  the  atomicity  property  and  prove  correctness  of  the  object 
implementations  by  establishing  a  simulation  mapping  between  the  im¬ 
plementation  and  the  specification  automata.  We  demonstrate  the  gen¬ 
erality  of  our  specification  by  showing  that  it  is  implemented  by  three 
different  read/write  register  constructions:  the  message-passing  register 
emulation  of  Attiya,  Bar-Noy  and  Dolev,  its  optimized  version  based  on 
real  time,  and  the  shared  memory  register  construction  of  Vitanyi  and 
Awerbuch.  In  addition,  we  show  that  a  simplified  version  of  our  specifi¬ 
cation  is  implemented  by  a  general  atomic  object  construction  based  on 
the  Lamport’s  replicated  state  machine  algorithm. 


1  Introduction 

Many  distributed  and  network-based  services  can  be  modeled  as  shared  objects 
accessible  to  (possibly  remote)  clients  through  well-defined  interfaces.  Atomicity 
[16,  21]  (also  known  as  linearizability  [10])  is  a  desirable  property  for  such  objects 
as  it  allows  clients  using  the  objects  to  perceive  the  operations  that  occur  in  each 
run  as  occurring  atomically,  in  some  sequential  order.  This  perception  makes  it 
easier  to  understand  the  behavior  of  a  system  using  distributed  services,  and  so, 
simplifies  the  task  of  system  design. 

Atomic  services  could  be  implemented  simply  on  single  server  machines.  How¬ 
ever,  to  achieve  high  availability  in  a  distributed  system  and  to  tolerate  failures, 
atomic  services  are  typically  implemented  by  distributed  algorithms.  Many  dis¬ 
tributed  algorithms  have  been  proposed  for  implementing  atomic  objects;  see, 
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for  example,  [17,15,36,27,33,32,10,35,14,19,18,22,23,8].  These  use  a  range 
of  techniques  to  achieve  the  appearance  of  total  ordering,  for  example,  assign¬ 
ing  timestamps  and  processing  operations  in  timestamp  order,  or  using  quorum 
configurations. 

Although  atomic  object  implementations  are  abundant,  proving  that  algo¬ 
rithms  achieve  atomicity  has  turned  out  to  be  a  challenging  problem.  Most  exist¬ 
ing  proofs  for  such  algorithms  are  long,  subtle,  and  difficult  to  understand  and 
check.  As  evidence  of  the  difficulty,  we  note  that  several  published  proofs  for 
implementations  of  atomic  shared  read/ write  memory  objects  have  later  been 
shown  to  be  incorrect.  We  believe  that  a  fundamental  reason  for  the  difficulty  of 
these  proofs  is  their  style:  they  are  based  on  detailed,  not- very-systematic,  rea¬ 
soning  about  events  and  their  ordering.  Useful  structure  in  such  proofs  is  often 
provided  by  lemmas  about  partial  orders  of  operations  on  objects,  for  example, 
Proposition  3  of  [16]  (for  single-writer  read/write  objects)  and  Lemma  13.16  of 
[21]  (for  multi-writer  read/write  objects).  These  lemmas  provide  sufficient  con¬ 
ditions  for  correctness  of  atomic  read/write  object  implementations,  based  on 
a  list  of  properties  that  a  partial  ordering  of  operations  must  satisfy.  However, 
showing  that  these  properties  hold  still  requires  detailed,  ad  hoc  reasoning  about 
events  (see,  e.g.,  [22,23]). 

In  this  paper,  we  study  systematic  ways  of  verifying  distributed  implemen¬ 
tations  of  atomic  objects,  beginning  with  read/write  objects  (registers).  Our 
general  approach  is  to  replace  operational  reasoning  about  events  and  partial 
orders  with  assertional  reasoning  about  invariants  and  simulation  relations.  The 
assertional  methods  differ  from  the  traditional  operational  arguments  in  two 
important  ways.  First,  the  system  properties  are  stated  precisely  in  terms  of 
predicates  over  the  system  state  components.  Second,  assertional  proofs  can  be 
checked  by  examining  individual  state  transitions  of  the  algorithm  without  rea¬ 
soning  about  entire  executions.  As  such  they  lend  themselves  to  mechanization, 
i.e.,  the  process  of  checking  a  proof  can  be  carried  out  using  interactive  tools, 
such  as  theorem  provers. 

Our  approach  to  carrying  out  assertional  atomicity  proofs  is  first  to  define 
an  abstract  state  machine  that  captures  the  atomicity  property  and  then,  prove 
correctness  of  the  object  implementations  by  establishing  a  simulation  mapping 
between  the  implementation  and  the  specification  automata.  The  challenge  is 
to  find  a  specification  automaton  that  is  general  enough  to  apply  to  many  ex¬ 
isting  implementations,  and  at  the  same  time  sufficiently  close  to  the  actual 
implementations  to  simplify  the  task  of  finding  the  mapping.  One  example  of 
an  atomicity  specification  that  turned  out  to  be  too  abstract  for  carrying  out 
simulation  proofs  is  the  canonical  atomic  object  automaton  of  Section  13.1.2 
of  [21].  The  canonical  object  automaton  maintains  a  buffer  used  to  store  incom¬ 
ing  client  requests.  Buffered  requests  can  later  be  applied  to  the  object  state, 
and  the  generated  responses  are  returned  to  their  originators.  Unfortunately, 
this  specification,  though  simple,  does  not  provide  sufficient  detail  to  allow  for 
easy  match  with  concrete  implementations. 


We  therefore,  give  more  detailed  specifications.  Namely,  we  define  an  ab¬ 
stract  state  machine,  which  we  call  the  Partial-Order  Machine  (PO-Machine), 
which  records  information  about  operations  and  their  orders  in  its  state.  The 
PO-Machine  expresses  the  common  behavior  of  many  existing  atomic  register 
implementations,  in  which  client  operation  requests  are  gradually  ordered  rela¬ 
tive  to  other  operation  requests  until  all  the  necessary  ordering  constraints  are 
achieved.  The  ordering  constructed  is,  in  the  limit,  guaranteed  to  be  a  partial 
order  of  the  requested  operations  that  satisfies  sufficient  conditions  for  showing 
atomicity. 

We  use  the  PO-Machine  as  a  formal  specification  for  distributed  algorithms 
that  implement  atomic  memory.  We  show  that  it  is  implemented  by  three  dif¬ 
ferent  read/write  register  constructions:  the  message-passing  emulation  of  At- 
tiya,  Bar-Noy,  and  Dolev  (ABD)  [3]  (extended  to  handle  multiple  writers  as  in 
[23]),  an  optimized  version  of  ABD  that  takes  advantage  of  synchronized  clocks 
at  writers  [8],  and  the  unbounded  version  of  the  shared  memory  construction 
of  a  multi- writer /multi-reader  register  from  single- writer/single-reader  registers 
of  [36].  We  also  show  that  a  slight  modification  of  the  PO-Machine,  called  the 
TO-Machine,  can  be  used  to  prove  atomicity  of  a  general  (i.e.,  not  necessarily 
read/write)  object  implementation  based  on  the  replicated  state  machine  proto¬ 
col  of  Lamport  [15]. 

We  specify  the  PO-Machine  and  the  algorithms  formally  using  the  I/O  Au¬ 
tomata  (IOA)[20]  and  Timed  IOA  [12,11]  models,  in  fact,  using  formal  spec¬ 
ification  languages  that  have  been  defined  for  these  models.  The  IOA/TIOA 
specification  languages  lead  to  very  stylized  assertional  proofs  for  invariants  and 
simulation  relations  that  can  be  partially  automated  using  theorem  provers. 
Moreover,  the  same  IOA  specifications  can  be  used  by  the  IOA  compiler  [31,  30] 
to  produce  executable  Java  code. 

Other  related  work:  Our  use  of  a  partial  order  automaton  as  an  abstract  spec¬ 
ification  was  inspired  by  prior  work  of  Fekete  et  al.  on  specifying  the  behavior 
of  an  Eventually  Serializable  Data  Service  [9].  Their  specification  used  a  (dif¬ 
ferent)  partial-order  machine,  which  expresses  weaker  consistency  requirements 
than  atomicity.  The  algorithm  studied  in  [9],  based  on  an  earlier  algorithm  of 
Liskov  et  al.  [13],  was  shown  to  achieve  this  weaker  form  of  consistency. 

The  only  other  published  simulation-based  atomicity  proofs  we  are  aware  of 
are  those  of  Bogdanov  [5]  (replicated  state  machine),  and  Doherty  et  al.  (lock- 
free  queue)  [7].  The  proofs  in  both  these  papers  are  complicated:  They  involve 
multiple  levels  of  asbtraction  as  well  as  both  forward  and  backward  simulations. 
In  contrast,  every  construction  considered  in  this  paper  is  shown  to  be  atomic 
by  exhibiting  a  single  forward  simulation  directly  from  the  implementation  au¬ 
tomaton  to  a  specification  automaton. 

Another  example  of  using  assertional  reasoning  for  proving  atomicity  is  the 
work  by  Wang  and  Stoller  [37] ,  which  uses  static  analysis  combined  with  model 
checking  to  verify  atomicity  of  code  blocks  involving  lock-free  synchronization 
primitives.  A  more  general  discussion  of  assertional  proof  techniques  can  be 
found  in  [28] . 


The  rest  of  the  paper  is  organized  as  follows:  In  Section  2,  we  introduce 
preliminary  definitions  and  notation  used  throughout  the  paper.  The  sufficient 
condition  for  proving  atomicity  is  specified  in  Section  3.  The  P  O-Machine  is 
described  in  Section  4.  The  ABD  algorithm  is  presented  and  proved  correct  in 
Section  5.  A  time-based  version  of  ABD  is  discussed  in  Section  6.  Section  7 
briefly  discusses  the  proofs  of  the  Vitanyi-Awerbuch’s  register  construction,  and 
of  the  Lamport’s  replicated  state  machine.  Section  8  discusses  future  directions. 
For  lack  of  space,  we  only  outline  intuition  and  highlight  basic  ideas  underlying 
the  correctness  proofs.  The  detailed  proofs  can  be  found  in  the  full  version  of 
the  paper  [6]. 

2  Preliminary  Definitions 

We  use  the  I/O  Automata  (IOA)[20]  model  to  formally  specify  services,  describe 
algorithms  and  carry  out  proofs.  An  I/O  automaton  is  a  non-determenistic  state 
machine  whose  state  can  change  atomically  through  a  discrete  transition  labeled 
by  a  discrete  action.  The  set  of  the  automaton’s  actions  is  called  the  action 
signature  of  the  automaton.  The  actions  can  be  either  external  or  internal.  The 
external  actions,  which  can  be  either  input  or  output,  model  interaction  with 
the  automaton’s  environment;  and  the  internal  actions  model  local  computation 
steps.  In  Section  6,  we  also  use  the  Timed  I/O  Automata  (TIOA)  model  [12, 
11],  which,  in  addition  to  discrete  transitions,  also  allows  the  automata  state  to 
evolve  by  trajectories ,  which  describe  evolution  of  the  state  over  time. 

We  use  forward  simulations  to  carry  out  atomicity  proofs.  Informally,  a  for¬ 
ward  simulation  is  a  relationship  between  the  states  of  two  automata  requiring 
that  the  transitions  of  one  system  can  in  some  sense  be  mimicked  by  the  other. 
A  precise  definition  of  the  simulation  formalism  can  be  found  in  [21] . 

The  read/write  service  A  read/write  object  (a  register )  type  consists  of  the 
following  components:  (1)  an  arbitrary  set  of  values  V  with  an  initial  value  Vo, 
(2)  the  set  of  operations  of  the  form  write(v),  v  €  V,  and  read,  (3)  the  set  of 
responses  are  ack  and  v  £  V,  and  (4)  the  sequential  specification  /  such  that 
f (w ,  write{v))  =  ( v,ack )  and  f(w,read)  =  (w,w). 

A  read/write  service  implements  a  shared  read/write  register.  To  access  the 
service,  a  client  issues  an  operation  descriptor  consisting  of  a  location  identifier 
loc,  and  an  operation  identifier  id.  In  addition,  the  write  operation  descriptor 
also  contains  a  value  val.  We  often  refer  to  an  operations  descriptor  x  simply 
as  operation  x,  and  denote  its  various  components  by  x.loc,  x.id ,  and  x.val.  We 
denote  by  Ow  and  Or  the  sets  of  the  write  and  the  read  operations  respectively, 
and  by  O  =  Ow  U  Or  the  set  of  all  operations.  For  a  set  X  C  O ,  we  denote  by 
X.id  =  {x.id  :  x  £  X}  the  set  of  identifiers  of  operations  in  X. 

Clients  use  the  actions  of  the  form  request(:r),  x  £  O,  and  response(x,  v),  x  £ 
O,  v  €  V U  {ack},  to  issue  operation  requests  and  receive  responses  respectively. 
Given  a  sequence  /?  of  the  request  and  response  actions,  an  requested  operation 
x  is  said  to  be  complete  in  j3  if  (3  contains  response(;r,  v)  for  some  v  £  V  U  {ack} 
which  we  call  the  return  value  of  x. 


We  say  that  (3  is  well-formed  if  there  exists  a  function  cause  mapping  each 
response  event  to  a  preceding  request  event  in  (3  so  that  the  following  is  satisfied: 
(1)  For  each  response  event  e  =  response(;r,  *),  cause(e)  =  request(a’)  (i.e.,  re¬ 
sponses  are  not  spuriously  generated);  and  (2)  cause  is  one-to-one  (i.e.,  responses 
are  not  duplicated)1. 

The  following  definition  will  be  used  throughout  the  paper:  Let  II  be  a  set  of 
read  and  write  operations,  and  R  be  a  binary  relation  over  II.  For  an  operation 
7r  £  77  we  define  last-prec-writes(TT ,  R)  =  {uj  £  Ow  :  £  R  A  flu/  €  Ow  : 

( u>,u >')  £  R  A  (u/,7r)  £  R}. 

3  Atomicity 

Atomicity  (or  linearizability)  is  specified  as  a  property  satisfied  by  the  object 
implementation  traces.  It  is  typically  defined  in  terms  of  the  existence  of  serial¬ 
ization  points  for  operations  so  that  shrinking  the  operations  to  occur  at  their 
serialization  points  results  in  a  valid  sequential  execution  of  the  read/ write  reg¬ 
ister  (see,  e.g.,  Chapter  13  of  [21],  Chapter  9  of  [4],  or  [10]).  For  our  purposes  in 
this  paper,  it  is  enough  to  give  a  sufficient  condition  for  proving  atomicity;  this 
condition  is  equivalent  to  the  one  in  Lemma  13.16  of  [21]. 

Let  (3  be  a  well- formed  sequence  of  the  actions  of  the  read/write  service 
interface  that  contains  no  incomplete  operations,  and  II  be  the  set  of  opera¬ 
tions  requested  in  (3.  We  say  that  (3  satisfies  Partial  Order  property  (henceforth, 
referred  to  as  PO)  if  there  exists  an  irreflexive  partial  ordering  -<  of  all  the 
operations  in  77,  satisfying  the  following: 

Property  1  (PO  Constraints) 

1.  If  the  response  event  for  tt  precedes  the  request  event  for  f>  in  (3,  then  4>  -ft  tt. 

2.  For  any  two  write  operations  it  and  (j>  in  77,  either  tt  -<  <fi  or  (j>  -<  tt. 

3.  If  tt  is  a  write  operation  in  PI  and  cf  is  a  read  operation  in  77  whose  request 
event  follows  the  response  event  for  tt,  then  tt  -<  4>. 

f.  If  tt  is  a  read  operation  in  77  and  (f>  is  a  read  operation  whose  request  event 
follows  the  response  event  for  tt,  then  for  each  u>  €  last-prec- writes { tt,<), 
ui  -<  </>. 

5.  Let  7 t  be  a  read  operation  in  77,  and  v  be  the  value  returned  by  tt.  If 
last-prec-writes(rr ,  -<;)  0,  then  v  =  uj.val  for  some 

lo  £  last-prec-writes(rr ,  -<;) .  Otherwise,  v  =  i>o. 

The  following  lemma  is  proved  in  [6] : 

Lemma  1.  (3  satisfies  PO  iff  there  exists  an  irreflexive  partial  ordering  of  all 
the  operations  in  77,  satisfying  the  (more  restrictive)  constraints  of  Lemma  13.16 
of  [21]. 

From  the  above  result  and  Lemma  13.16  of  [21],  we  obtain: 

Lemma  2.  If  (3  is  well-formed  and  satisfies  PO,  then  (3  satisfies  atomicity. 

1  Note  that  our  notion  of  well  formedness  is  weaker  than  that  usually  found  in  the 
literature  as  it  allows  requests  from  the  same  location  to  be  issued  concurrently. 


4  The  PO-Machine 


In  this  section  we  define  the  Partial-Order  Machine.  First,  we  formally  specify 
the  environment  assumptions  of  the  read/write  service.  This  environment  is 
represented  by  a  single  automaton,  called  Users,  whose  code  could  be  found 
in  [6].  The  Users  automaton  contains  a  single  variable  requested  to  keep  track 
of  the  ids  of  requested  operations,  in  order  to  avoid  repeats.  An  implementation 
of  the  environment  would  not  have  such  a  variable,  but  would  use  some  other 
mechanism  to  ensure  unique  operation  ids  (e.g.,  client  id  and  a  counter). 

Lemma  3.  For  x,y  £  requested,  x  =  y  <t=>  x.id  =  y.id. 

The  PO-Machine  signature  and  state  variables  appear  in  Figure  1,  and  its 
transitions  appear  in  Figure  2.  This  automaton  maintains  a  partial  order  in 
its  state,  represented  by  variables  vertices  and  edges.  Vertices  correspond  to 
requested  operations,  and  edges  to  ordering  relationships  that  have  been  deter¬ 
mined  for  these  operations.  When  a  request  arrives,  it  is  put  into  vertices ;  later, 
it  becomes  classified  as  ordered,  then  completed ,  and  finally,  responded.  Edges 
may  be  added  at  any  time  from  ordered  write  operations  to  unordered  ones  (see 
action  add-edge). 

An  unordered  operation  7r  may  become  ordered  at  any  time  after  it  has 
acquired  incoming  edges  from  all  write  operations  that  completed  before  tt  began 
(i.e.,  all  writes  in  prec( 7r)).  This  ensures  that  constraints  1  and  3  of  Property  1 
hold  among  all  writes,  and  between  writes  and  reads.  Constraint  1  is  also  trivially 
preserved  among  reads  as  edges  originating  at  read  requests  are  disallowed  by 
the  PO  signature  (see  Figure  1).  When  a  write  operation  n  becomes  ordered, 
new  edges  are  inserted  to  ensure  that  7r  is  ordered  with  respect  to  all  previously- 
ordered  write  operations  (see  action  order)  so  that  constraint  2  of  Property  1  is 
satisfied. 

An  ordered  operation  may  become  completed  at  any  time;  when  a  read  oper¬ 
ation  (j>  completes,  it  also  forces  each  write  operation  7r  immediately  preceding  4> 
in  the  partial  order  to  complete.  This  ensures  that  every  read  operation  invoked 
after  <fi  completes  will  find  7r  in  its  prec  set,  and  will  therefore,  become  ordered 
only  after  it  has  an  incoming  edge  from  n.  This  guarantees  that  constraint  4  of 
Property  1  is  satisfied,  and  also  captures  the  essence  of  the  “helping”  mechanism 
found  in  many  atomic  register  implementations. 

A  completed  operation  is  allowed  to  return  a  response.  The  response  returned 
by  a  read  operation  is  the  value  written  by  the  last  preceding  (in  the  partial 
order)  write  operation,  or  the  initial  value  if  no  such  write  exists  (see  action 
response).  Thus,  constraint  5  of  Property  1  is  satisfied. 

In  [6],  we  prove  that  the  limit  of  the  transitive  closure  of  (vertices,  edges), 
maintained  in  the  derived  variable  dag ,  satisfies  Property  1.  Since  every  trace 
of  PO-Machine  is  obviously  well-formed,  by  Lemma  2,  PO-Machine  implements 
an  atomic  register: 

Theorem  1.  Each  trace  of  the  PO-Machine  satisfies  atomicity. 


Input: 

request(x),  x  G  CD 


Output: 

response(x,  v),  x  G  CD, 
v  G  V  U  { acfc } 


Internal: 

add-edge (x,  y),  x  G  CDW,  y  G  CDW  U  Or 
order (x),  x  G  CD 
complete (x),  x  £  O 


State: 


vertices  C  CD,  initially  empty 
ordered  C  CD,  initially  empty 
completed  C  CD,  initially  empty 


responded  C  CD,  initially  empty 
edges  C  CD  X  CD ,  initially  empty 

prec  is  a  partial  function  from  CD  to  subsets  of  CD, 
initially  empty 


Derived  vars: 

dag,  the  transitive  closure  of  (vertices ,  edges') 


For  x  £  CDr  ,  last- writes (x)  =  last-prec- writes (x ,  dag ) 


Fig.  1.  PO-Machine  signature  and  states 

Input  request(x) 

Internal  complete(x) 

Effect: 

Precondition: 

vertices  :=  vertices  U  {x} 

x  G  ordered  —  completed 

prec(x)  :=  completed  f~l  (Dw 

Effect: 

completed  :=  completed  U  {x} 
if  x  G  Ot  then 

Internal  add-edge(x,  y) 

Vy  G  last- writes (x)  do 

Precondition: 

completed  :=  completed  U  -fy} 

y  G  vertices  —  ordered 
x  G  ordered 

Effect: 

Output  response(x,  ack),  x  G  CDW 

edges  :=  edges  U  {( x ,  y)} 

Precondition: 

x  G  completed  —  responded 

Effect: 

Internal  order(x),  x  G  CDW 

resvonded  :=  resvonded  U  -fxl- 

Precondition: 

x  G  vertices  —  ordered 

V y  G  prec(x)  :  (y,  x)  G  dag 

Output  response(x,  vq),  x  G  CD r 

Effect: 

Precondition: 

edges  :=  edges  U  {(x,  y)  :  y  G  ordered  l~l  CDW  A 

x  G  completed  —  responded 

( V , x)  £  dag} 

last- writes  (x)  =  0 

ordered  :=  ordered  U  {x} 

Effect: 

responded  :=  responded  U  {x} 

Internal  order(x),  x  G  CDr 

Precondition: 

Output  response(x,  v),  x  G  CDr 

x  G  vertices  —  ordered 

Precondition: 

Vy  G  prec(x)  :  (y,  x)  G  dag 

x  G  completed  —  responded 

Effect: 

last- writes (x)  ^  0 

ordered  :=  ordered  U  {x} 

v  =  w  .val  :  w  G  last- writes (x) 

Effect: 

responded  :=  responded  U  {x} 

Fig.  2.  PO-Machine  transitions 


5  The  Attiya,  Bar-Noy,  and  Dolev  Algorithm 

In  this  section,  we  present  a  distributed  wait-free  implementation  of  an  atomic 
multi- writer/multi-reader  register  based  on  the  well-known  message-passing  al¬ 
gorithm  of  Attiya,  Bar-Noy,  and  Dolev  [3]  (which  we  call  ABD).  We  prove  cor¬ 
rectness  of  ABD  by  showing  that  ABD  implements  PO-Machine,  which  by  The¬ 
orem  1,  implies  that  ABD  implements  an  atomic  register. 

The  original  ABD  protocol  implements  a  wait-free  atomic  read/write  regis¬ 
ter  using  a  collection  of  n  processes  communicating  among  themselves  through 
reliable  point-to-point  channels.  The  implementation  is  resilient  to  up  to  n/2  pro¬ 
cess  crashes.  Each  process  in  ABD  is  responsible  for  both:  handling  the  client 
operation  requests,  and  storing  and  updating  the  local  copy  of  the  register  value. 


Here,  we  present  a  generalized  version  of  ABD  where  we  let  the  two  roles  in 
the  ABD  protocol  be  performed  by  two  classes  of  agents:  clients  and  replicas. 
This  design  allows  for  flexibility  in  assigning  roles  to  actual  network  locations 
thus  simplifying  the  algorithm  deployment  in  real  systems.  We  also  use  a  sepa¬ 
rate  client  to  handle  each  user  request  so  that  the  actual  clients  can  handle  any 
number  of  requests  and  in  whatever  order  (for  example,  requests  can  be  par¬ 
titioned  among  several  threads,  or  executed  sequentially).  Our  implementation 
also  supports  multiple  writers  using  the  technique  of  [23] . 

We  now  describe  the  ABD  implementation  (the  ABD  automaton)  in  more 
detail.  Let  P  be  a  finite  set  of  replicas.  We  define  a  quorum  system  Q  on  P  to 
be  the  union  of  a  set  of  write  quorums  Qw  and  the  set  of  read  quorums  Qr. 
Qw  and  Qr  are  sets  of  subsets  of  P  such  that  for  each  Qw  G  Qw  and  Qr  G  Qr , 
Qw  fl  Qr  ^  0.  The  ABD  automaton  is  the  composition  of  the  Users  automaton 
of  Section  4,  the  client  automata  Cx,  x  G  O,  the  replica  automata  Rp,  p  G  P, 
and  the  reliable  point-to-point  channel  automata  connecting  each  client  Cx  with 
replica  Rp  and  vice  versa.  The  client’s  interface  and  state  variables  appear  in 
Figure  3.  The  code  of  the  reader  client,  the  writer  client  and  the  replica  appear 
in  Figures  4,  5,  and  6  respectively.  We  do  not  present  the  specification  for  the 
channel  automata  as  their  functionality  is  obvious. 

The  value  stored  at  each  replica  is  associated  with  a  tag.  Tags  are  two-field 
records  consisting  of  a  sequence  number  sn,  which  is  a  non-negative  integer,  and 
a  request  identifier  id.  Tags  are  ordered  lexicographically  with  the  precendence 
to  the  sequence  number  field. 

Clients  access  read  (resp.  write)  quorums  by  first  sending  a  message  to  all 
the  replicas,  and  then  awaiting  responses  from  a  write  (resp.  a  read)  quorum. 
The  request  handling  at  clients  involves  two  rounds  of  quorum  accesses,  called 
the  read  phase  and  the  write  phase  respectively,  such  that  a  read  quorum  is 
contacted  during  the  read  phase,  and  a  write  quorum  is  contacted  during  the 
write  phase.  A  client  keeps  track  of  the  request  progress  through  the  phases 
using  the  variable  status.  The  operation’s  status  is  initially  idle.  It  is  changed 
to  pending  (p)  at  the  beginning  of  the  read  phase.  It  becomes  sending  (s)  at  the 
beginning  of  the  write  phase.  It  is  changed  to  committed  (c)  upon  completion  of 
the  write  phase,  and  finally  to  responded  (r)  after  a  response  is  returned. 

Specifically,  to  handle  a  write  request  x,  the  client  Cx  (see  Figure  5)  performs 
a  read  phase  to  determine  the  highest  tag  t  associated  with  the  values  stored  at 
some  read  quorum.  It  then  performs  a  write  phase  to  store  the  value  v  associated 
with  tag  ( t.sn ,  x.id)  at  a  write  quorum.  It  then  responds  with  ack.  To  handle  a 
read  request  y ,  client  Cy  (see  Figure  4)  first  performs  a  read  phase  to  determine 
the  value  v  associated  with  the  highest  tag  t  among  those  associated  with  the 
values  stored  at  some  read  quorum.  It  then  performs  a  write  phase  to  guarantee 
that  the  pair  ( t ,  v)  is  stored  at  a  write  quorum.  It  then  responds  with  v. 

The  replica’s  algorithm  (see  Figure  6)  is  simple:  In  response  to  a  read  phase 
message,  a  replica  p  either  responds  with  its  current  tag  (for  write  requests),  or 
the  current  tag  and  the  value  (for  read  requests).  In  response  to  a  write  phase 
message  carrying  a  tag  which  is  bigger  than  p’s  current  tag,  p  overwrites  its 


current  tag  and  the  value  with  those  in  the  message.  Otherwise,  the  p’s  state  is 
left  unchanged.  In  both  cases,  p  responds  with  ack. 


Types: 

Tag  —  A/”  —  ®  X  O  .id,  with  selectors  sn  and  id,  ordered  lexicographically 
Phase  =  {idle,  p,  s,  c,  r},  ordered  so  that  idle  <p<s<c<r 


Signature: 

Input: 

request  (x) 

recei ve(m)p>x,  p  e  P,  m  e  {acfc}  U 
U  ( Tag  X  V) 


Output: 

response(x,  v),  v  G  V  U  {acfc} 
send(m)X)p,  p  £  P,  m  £  {r,  w}  U 
(Tag  X  V) 


State: 

status  G  Phase,  initially  idle  read-resp  G  P,  initially  empty 


val  G  V ,  initially  undefined  write-resp  G  P,  initilly  empty 

tag  G  Tag,  initially  (0,  iq)  for  each  p  G  P:  req-bufferp  G  seqof({r,  u>}  U 

(Tag  X  V)),  initially  A 


Fig.  3.  The  state  and  signature  of  client  automata  Cx,  x  £  O  for  ABD. 


Input  request(x) 

Effect: 

status  :=  p 
for  each  p  G  P- 

append  ( r )  to  req-bufferp 

Input  receive(t;,  t)PjX 
Effect: 

read-resp  :=  read-resp  U  { p } 
if  status  =  p  A  t  >  tag  then 
val  :=  v 
tag  t 

Internal  rq-collected(q)a; 

Precondition: 
status  =  p 
read-resp  3  q 
Effect: 

status  :=  s 
for  each  p  G  P- 

append  (tag,  val )  to  req-bufferp 


Input  receive(acfc)p)x 
Effect: 

write-resp  :  —  write-resp  U  { p } 

Internal  wq-collected(q)a; 
Precondition: 
status  =  s 
write-resp  D  q 
Effect: 


Output  response(x,  v) 
Precondition: 
status  =  c 
val  =  v 
Effect: 

status  r 

Output  send(m)a3,p 
Precondition: 

req- buffer p  ^  A 
m  =  head(req-bufferp ) 
Effect: 


delete  head  of  req-bufferp 


Fig.  4.  Transitions  of  reader  Cx,  x  £  Or  for  ABD. 


Correctness  of  ABD:  We  now  prove  that  ABD  implements  an  atomic  register. 
Our  strategy  will  be  to  show  that  ABD  implements  PO-Machine  by  exhibiting  a 
forward  simulation  from  ABD  to  PO-Machine.  In  the  following,  for  each  x  £  O , 
we  will  use  subscript  x  to  refer  to  the  state  variables  of  Cx.  It  is  convenient 
for  the  ABD  correctness  proof  to  define  several  derived  variables  for  the  ABD 
automaton.  These  are  summarized  in  Figure  7. 


Input  request(a;) 

Effect: 

status  :=  p 
for  each  p  £  P: 

append  ( w )  to  req-bufferp 

Input  receive(sn)pj£C ,  sn  £  A f  —  ® 

Effect: 

read-resp  :=  read-resp  U  { p } 
if  status  =  p  A  sn  >  tag.sn  then 
tag.sn  :=  sn 


Internal  rq-collected(q)a; 

Precondition: 
status  =  p 
read-resp  3  q 
Effect: 

status  :  =  s 

tag.sn  :=  tag.sn  +  1 

for  each  p  £  P: 

append  ( tag,x.val )  to  req-bufferp 


Input  receive(  acfc)p;X 
Effect: 

write-resp  :  =  write-resp  U  { p  } 


Internal  wq-collected(q)a; 

Precondition: 
status  =  s 
write-resp  D  q 

Effect: 

status  :  =  c 

Output  response(a;,  ack ) 

Precondition: 
status  =  c 

Effect: 

status  :=  r 


Output  send(m,)a3)p 
Precondition: 

m  =  head(req-bufferp) 


Effect: 

delete  head  of  req-buffe 


V 


Fig.  5.  Transitions  of  writer  Cx,  x  £  Ow  for  ABD. 


Signature: 

Input:  Output: 

receive ( m. ) x  )P  ,  x  G  O,  m  £  {r,  u;}  U  ( Tag  X  V)  send(m)p)X,  x  £  O,  p  €  R,  m  £  {acfc}  U  ( Tag  X  V) 

State: 

val  £  V,  initially  vq 
tag  e  Tag,  initially  (O.iq) 

For  each  x  £  O:  resp-bufferx  £  seqof  ({ack}  U  AT  —  ®  U  ( Tag  X  V)),  initially  A 


Transitions: 


Input  receive 

Input  receive(t,  v)x,p 

Output  send(m)p)X 

Effect: 

Effect: 

Precondition: 

append  (val,  tag)  to  resp-bufferx 

if  t  >  tag  then 

resp-buffer x  ^  A 

tag  :=  t 

m  =  head(resp-bufferx) 

Input  receive (io)x,p 

Effect: 

append  (tag .  sn)  to  resp-buffer ^ 

val  :=  v 

Effect: 

append  (ack)  to  resp-buffer x 

delete  head  of  resp-buffer. 

Fig.  6.  Replica  automaton  Rp,  p  £  P  for  ABD 


Among  these  variables,  the  most  interesting  one  is  min-tag  which  is  used  to 
keep  track  of  the  lowest  possible  tag  that  could  ever  be  determined  by  a  client  at 
the  end  of  the  read  phase.  At  the  beginning  and  before  any  replica  has  responded, 
min-tag  is  the  smallest  tag  among  the  maximum  tags  carried  by  replicas  in  every 
read  quorum.  As  the  client  is  progressing  through  the  read  phase  it  might  get  a 
response  from  a  replica  whose  tag  is  bigger  than  the  current  value  of  min-tag. 
In  this  case,  the  definition  of  min-tag  ensures  that  min-tag  is  assigned  to  that 
higher  value.  Finally,  upon  completion  of  the  read  phase,  the  value  of  min-tag 
is  fixed  to  be  the  maximum  tag  received  during  the  phase.  The  simulation  proof 
relies  on  the  following  key  property  of  min-tag : 

Lemma  4.  For  each  x  £  O,  min-tag  (x)  is  non- decreasing. 

The  simulation  mapping  from  the  states  of  ABD  to  the  states  of  the  PO- 
Machine  appears  in  Figure  8.  The  first  four  components  of  the  mapping  are 


-  pending  =  {x  £  O  :  statusx  >  p} 

-  ordered  -  { x  £  O  :  statusx  >  s} 

-  completed  =  {x  £  O  :  statusx  >  c } 

-  responded  =  {x  £  O  :  statusx  >  r} 

-  For  r  £  Or:  last- writes  (r)  =  {u>  £  Ow  Pi  ordered  :  s.tagw  =  s.tagr } 

-  For  x  £  O,  p  £  P: 

(  t,  if  3v  £  V  :  ( v,t )  £  resp-bufferpx  U  channelp,x 
new-tag (x ,  p)  =  <  (sn,  x.id),  if  ( sn )  £  resp-bufferp  ,x  U  channelp>x 
tagp,  otherwise 

-  For  x  £  O: 

C  maxftaga;,  min Q^Qr  ma.x{new-tag(x ,  p)  •  p  €  Q  \  read-resp x}\ , 
min-tag(x)  =  <  if  VQ  6  Qr,  read-resp  2  Q 

^  tagx  ,  otherwise 


Fig.  7.  Derived  variables  for  the  ABD  automaton 


straightforward:  All  the  operations  that  have  ever  been  requested  (indicated  by 
status  >  idle)  are  mapped  to  vertices ;  the  operations  that  have  completed  the 
read  phase  and  acquired  final  tags  (indicated  by  status  >  p)  are  mapped  to 
ordered ;  and  the  operations  that  have  responded  (indicated  by  status  >  c)  are 
mapped  to  responded. 

The  set  of  edges  consists  only  of  edges  among  operations  that  have  completed 
their  read  phases  (8.7).  The  edges  among  these  operations  are  determined  by 
their  tag  order  and  type.  Specifically,  any  two  writes  x  and  y,  such  that  tagx  < 
tagy,  are  connected  by  edge  (x,y)  (8.8);  and  each  read  x  and  write  y  such 
that  tagx  =  tagy ,  are  connected  through  edge  (y,x)  (8.9).  To  maintain  the 
mapping  for  edges,  each  rq-collected(a:)  for  x  €  Ow  is  simulated  by  a  sequence  of 
add-edge(y,  x)  for  each  ordered  write  operation  y  such  that  tagy  <  tagx,  followed 
by  order(a:);  and  each  rq-collected(x)  for  x  £  Ow  is  simulated  by  a  sequence  of 
add-edge(y,  x)  for  each  ordered  operation  y  such  that  tagy  =  tagx.  No  actions 
involving  unordered  operations  (i.e.,  the  operations  with  status  <  s)  result  in 
adding  new  edges. 


f  is  the  relation  over  states(PO  —  M achine)  X  states(AB D)  such  that  each  (s,  u)  £  f  iff: 

1.  u. requested  —  s  .requested 

2.  u.  vertices  =  s  .pending 

3.  u. ordered  —  s .ordered 

4.  u.  completed  =  s  .completed  U  I  I  s  .last-  writes  (r) 

Vr60rns.  completed 

5.  u. responded  =  s  .responded 

6.  For  all  x  £  u. vertices,  if  y  £  u.prec(s),  then  s.tagy  <  s.min-tag(x) 

7.  u.dag  C  s .  ordered  X  s .  ordered 

8.  For  all  x,  y  £  Ow  Pi  u  .ordered,  if  (x,  y )  £  u.dag,  then  s.tagx  <  s.tagy 

9.  For  all  x  £  O-w  fl  u. ordered  and  y  £  Or  PI  u. ordered,  (x,  y)  £  u. edges  iff  s.tagx  =  s.tagy 


Fig.  8.  Forward  simulation  from  ABD  to  PO-Machine 


The  most  interesting  part  of  the  proof  is  to  show  that  order  (a;)  becomes 
enabled  once  all  the  (y,  x)  edges  have  been  added.  For  that  we  need  to  show 
that  the  tag  acquired  by  x  at  the  end  of  the  read  phase  is  at  least  as  big  as 
the  tag  of  every  operation  that  had  completed  before  x  began.  Since  at  the 


end  of  the  read  phase,  tagx  =  min-tag(x),  the  necessary  enabling  condition  is 
provided  by  part  8.6  of  the  mapping  that  requires  that  for  each  y  £  prec(x), 
tagy  <  min-tag{x). 

To  show  that  8.6  is  maintained  throughout  the  read  phase  of  x,  request(x) 
is  simulated  by  the  request(:r)  action  of  the  PO-Machine;  and  each  receive  is 
simulated  by  the  empty  sequence.  Since  at  the  time  x  is  invoked,  the  tag  of  every 
■y  €  prec{x)  has  been  stored  at  a  write  quorum  of  replicas,  and  because  every  pair 
of  write  and  read  quorums  intersects,  minQegr  ma xp£Q{tagp}  >  tagy.  Hence,  8.6 
is  preserved  by  request(x).  Finally,  since  min-tag( x)  is  non-decreasing  (Lemma  4) 
and  prec{x)  is  not  affected  by  any  action  except  request,  8.6  is  preserved  by 
receive.  Hence,  by  the  end  of  the  read  phase  of  x,  for  each  y  £  prec(x),  tagy  < 
min-tag(x)  as  required. 

We  argued  informally  that  the  mapping  in  Figure  8  is  a  forward  simulation 
from  ABD  to  the  PO-Machine.  A  detailed  proof  appears  in  [6]. 

Lemma  5.  The  mapping  in  Figure  8  is  a  forward  simulation  from  ABD  to  the 
PO-Machine. 

Since  by  Theorem  1,  each  trace  of  the  PO-Machine  satisfies  atomicity,  the 
same  is  true  for  every  trace  of  ABD: 

Theorem  2.  Each  trace  of  ABD  satisfies  atomicity. 

Automated  Tools  Support:  We  have  used  the  TIOA  to  PVS  translator  and  TAME 
library  [2]  to  generate  descriptions  of  the  PO-Machine  and  the  ABD  algorithm 
in  the  language  of  the  Prototype  Verification  System  (PVS)  [26].  We  used  PVS 
to  substantially  increase  the  level  of  detail  and  assurance  of  some  of  our  previous 
hand  proofs.  In  fact,  we  discovered  several  gaps  and  bugs  in  our  hand  proofs. 
Automatic  translation  enabled  us  to  easily  tweak  the  simulation  relations  and 
rerun  the  proof  scripts.  We  also  used  the  IOA  code  generator  tool  [31,30]  to 
compile  the  verified  ABD  automaton  into  an  executable  Java  code.  This  way, 
a  single  formal  representation  of  the  ABD  algorithm  was  used  for  specification, 
verification,  and  execution. 

6  Timed  ABD 

In  this  section,  we  present  an  optimized  version  of  the  ABD  protocol,  called 
Timed-ABD,  that  takes  advantage  of  perfectly  synchronized  clocks  at  the  writers 
to  eliminate  the  read  phase  of  the  write  implementation  (see  [8]). 

The  Timed-ABD  is  the  composition  of  the  following  timed  automata:  the 
replica  and  reader  client  automata  in  Figures  6  and  4  respectively  augmented 
with  arbitrary  trajectories  that  keep  their  state  unchanged;  and  the  writer  client 
automata  whose  code  appears  in  Figure  9.  To  model  synchronized  clocks,  each 
writer  maintains  a  local  variable  clock  whose  trajectory  is  d(clock)  =  1  (i.e.,  the 
clock  value  grows  continuously,  at  the  same  rate  as  the  real  time). 

The  writer  algorithm  is  as  follows:  To  write  a  value,  the  writer  first  takes  its 
current  clock  reading,  and  then  delays  its  execution  until  its  clock  exceeds  the 


Signatu 


Input: 

request  (x)^ 

receive(m)p)X,  p  G  P,  m  G  {ack} 


Internal: 

orderx 

j\f>0  wq-col lected ( q ) x  >  g  £  Q 


Output: 

response(x,  v),  v  G  { ack } 
send(m,)X)p ,  m  G  {«;}  U  ( Tag  X  V) 


State: 

clock  G  7?.,  initially  0 
Discrete  req-time  G  7?.,  initially  0 
status  G  Phase,  initially  idle 


Transitions: 

Input  request(a:) 

Effect: 

status  :=  p 
req-time  :=  clock 

Internal  order^ 

Precondition: 

clock  >  req-time 
status  =  p 
Effect: 

tag.sn  :=  clock 
status  s 
for  each  p  G  P  ■ 

append  ( tag,x.val )  to  req-bufferp 


Trajectories: 

evolve 

d(clock)  =  1 

All  the  other  state  variables  are  kept  unchanged 


tag  G  Tag,  initially  (0,  x.id ) 
write-resp  C  P,  initilly  empty 

for  each  p  G  P:  req-bufferp  G  seqo/({u>}  U  {Tag  X  V)), 
initially  A 


Input  receive(acfc)p>;E 
Effect: 

write-resp  :=  write-resp  U  {p} 

Internal  wq-col  lected  (q)x 
Precondition: 
status  =  s 
write-resp  D  q 
Effect: 

status  :=  c 

Output  response(x,  ack) 
Precondition: 

status  =  c 
Effect: 

status  r 

Output  send(m)a3)p 
Precondition: 

m  =  head(req-bufferp) 
Effect: 

delete  head  of  req-bufferp 


Fig.  9.  Writer  client  Cx,  x  £  Ow  for  Timed- ABD 


initial  reading.  The  second  clock  reading  is  used  as  the  tag  with  which  the  client 
performs  the  write  phase. 

The  simulation  mapping  from  the  states  of  Timed-ABD  to  the  states  of 
Timed-PO  (i.e.,  the  PO-Machine  augmented  with  arbitrary  trajectories  that  do 
not  change  its  state)  appears  in  Figure  10.  To  see  that  the  mapping  is  preserved, 
we  observe  that  a  write  operation  becomes  ordered  once  it  is  verified  that  a 
non-zero  amount  of  time  has  elapsed  since  it  was  requested.  We  therefore,  simu¬ 
late  each  Timed-ABD  trajectory  corresponding  to  a  non-zero  time  interval  by  a 
trajectory  of  Timed-PO  of  the  same  length,  followed  by  a  sequence  of  add-edge 
actions,  followed  by  order.  The  rest  of  the  simulation  proof  is  straightforward 
(see  [6]  for  details). 


f  is  the  relation  over  states  {Timed- ABD)  X  states  {Timed-PO)  such  that  {s ,  u)  G  /  iff: 


1-5: 

Identical 

to  1-5 

n  Figure  8 

6: 

For  all  x 

G  u.ve 

rtices  n  Or, 

if  y  G  u ,prec{x) ,  then 

s.tagy  <  s.min-tag{x) 

7-8: 

Identical 

to  7-8 

n  Figure  8 

9: 

For  all  x 

G  u.ve 

rtices  n  Ow 

if  y  G  u  ,prec{x) ,  ther 

s.tagy.sn  <  s.req-timex 

10: 

For  all  x 

6  (u.i 

ertices  —  u. 

ordered)  D  (Dw ,  y  G 

i.  ordered  f~l  Ow,  if  s  .tagy  .  sn  <  s  ,clockx  ,  then  {y ,  x)  G 

u. edges. 


Fig.  10.  Forward  simulation  from  Timed-ABD  to  Timed-PO 


7  Other  Algorithms 


We  discuss  briefly  how  to  prove  atomicity  of  the  unbounded  multi- writer /multi¬ 
reader  register  construction  of  Vitanyi  and  Awerbuch  [36]  (referred  to  henceforth 
as  VA),  and  of  a  general  atomic  object  implementation  based  on  the  replicated 
state  machine  algorithm  of  Lamport  [15]  (referred  to  henceforth  as  RSM). 

First,  we  observe  that  VA  can  be  recast  as  a  special  case  of  ABD  with  the 
write  quorums  being  the  rows  and  the  read  quorums  being  the  collumns  of  the 
matrix.  Therefore,  the  simulation  proof  of  VA  is  almost  identical  to  that  of  ABD. 
In  particular,  it  is  easy  to  see  that  the  simulation  from  ABD  to  PO-Machine  in 
Figure  8  is  also  a  forward  simulation  from  VA  to  PO-Machine. 

To  prove  atomicity  of  RSM,  we  use  a  simplified  version  of  the  PO-Machine, 
called  TO-Machine.  The  TO-Machine  constructs  a  single  total  order  of  all  the 
requested  operations.  In  particular,  every  operation  becomes  ordered  only  after 
it  is  ordered  relative  to  all  the  other  ordered  operations.  The  TO-Machine  is 
parameterized  by  the  emulated  object  sequential  specification  and  initial  state 
which  are  used  to  compute  responses.  The  simulation  proof  is  based  on  the  ob¬ 
servation  that  in  RSM,  an  operation  x  becomes  ordered  once  the  local  timestamp 
at  each  replica  becomes  greater  than  that  of  x.  The  full  proof  appears  in  [6] . 

8  Conclusions  and  Future  Work 

Our  work  with  four  algorithms  so  far  suggests  to  us  that  our  PO-Machine  (or 
small  variants)  may  be  general  enough  to  capture  many  of  the  existing  atomic 
register  algorithms.  We  plan  to  use  these  methods  to  study  a  wider  variety  of  al¬ 
gorithms,  such  as  bounded-timestamp-based  constructions  (see  e.g.,  [34]),  whose 
proofs  have  been  notoriously  difficult  and  bug-prone.  An  interesting  challenge 
will  be  to  extend  our  framework  to  capture  implementations  that  are  not  ex¬ 
plicitly  based  on  timestamps,  for  example,  the  construction  that  creates  atomic 
bits  from  safe  bits  [32],  Another  interesting  direction  deals  with  adapting  the 
PO-Machine  to  capture  weaker  register  semantics,  such  as  safe  registers,  reg¬ 
ular  registers  (including  the  multi- writer  regular  registers  of  Welch  [29]),  and 
sequentially  consistent  registers.  There  is  an  increased  recent  interest  in  these 
semantics  as  they  capture  the  guarantees  provided  by  many  Byzantine-resilient 
storage  systems  [24,  25, 1]  based  on  Byzantine  quorums  [24] . 

Yet  another  interesting  application  domain  for  our  techniques  is  the  verifi¬ 
cation  of  multi-threaded  programs  based  on  lock-free  synchronization  primitives 
(such  as  CAS,  LL/SC,  etc.).  This  area  has  recently  been  receiving  an  increased 
attention  due  to  the  growing  popularity  of  multi-processor  computing  platforms, 
and  the  introduction  of  lock- free  synchronization  primitives  into  the  Java  con¬ 
currency  package. 

Finally,  we  are  interested  in  identifying  common  patterns  behind  many  di¬ 
verse  implementations  of  atomic  objects.  This  will  make  it  easier  to  understand 
and  compare  different  algorithms.  We  expect  that  such  patterns  should  be  ex¬ 
pressible  in  terms  of  common  specification  automata  (e.g.,  a  unified  version  of 
the  PO-  and  TO-Machines). 
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